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The purpose of this paper is to present an applied sodel 
* 

for the utilization, of eraltiatioh in the process, of develop- 
ing instructional' staterials . The laodel vill be introduced 
throtteh 5t pr-esen-fcrtioa -of izs concep ttJAl r&ti o n& Sre/ e 



tion of it6 Htetbodology and instrumentation / and a deaonstfci- 
. tion of irs developTnent and utilization durmg a three-^yeax 
period vitiixn the framevorX of a project of curriculum devel- 
(XCTent at the Tel -Aviv University. < 
The evaluation xaodel presented in this- paper is based on 

a definition of evaluation derived fr o ca sc^g _ttieoret>ical prop-: 

2 * 3 ' ! 

ositions suggested by Stuff l€i>eani and by Screven. The tens * 

"evaluation* will be used herein to refer= to |the process of 
delineating, obtaining, ""and providing information on^^the laerit 
of goals, designs > ixaplesientation / and outcoiaes o& (education- 
al) activities, to serve formative and smaaative- purposes . 
This definition is ccsrajrised of fourSaajor elements . The 
first elesaent suggests that the process of evaluation should 

— ET^ ' ^ ■ 

. "The author would like to acknwledge the contribution of 

his colleagues at the HBTA Project during the period 1970-1972. 

The4:r positive as well as negative attitudes towards evaluation 

provided an important source of inspiration for the d^velopiaent 

of this evaluation laodel. 

2 ' ' ' 

Daniel X. SJtuf f lebean, et al., Bducation£tl Bvstluation and 

decision Making , Itasca/ 111.: Peacock, 1971. 

3 . ' » J * 

Michaiel Scriven, "The Methodology of Bval<iatioS?L.. In 
Etobert E . Stake (Ed . ) , Curriculua Evaluation/ ABRft Mon^raph 
Series on Evaluation, Do. 1 Chicago: Rand KcNally, l$w^^ 
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not be lindted to the rechnical activities of itkeasuremeist apa 
data gathering {"obtaining") / irat should also, include actavi- 
ties intended to identify evalxiaxion needs ( •delin^tisg" ) 
aiw3 provide th© obtained information to the re>evan^* atidiences 
of the evaluation., The second el^i^t pf. thl5_Jl^iJiitiQn 



pli^rS thar evaluation concerns info grriat ion on^ the as&esg7i>ent 
of izezrit / in an atterot to detenaina the vp^th or value of 



educational endeavors- The thi^ eLament m the definition 
suggests that judgment of xaerit shoiald not be Imated to ou 



oo^s of . educational activities bu^ should also address ques- 
tions related to the merit of thfiit goals / their designs >and 
their isg)leiaentation . The fourth Bleiaent in the definitibn 
reflects Scriven*s distinction beti^een fpnaative and gi?T7gna^ 
tive evaluation suggesting that evaluation should help to 
iinprove an educational product during the process of its de- 
velopment/ and/or aemonstrate the zaerit of the final product 
when its. development is co25>leted. Howfever, the focus of the 
sxaggested evaltiation raodel is on the forxaative use of evkiu- 

ation in curricixlma development/ intended to improve the.'de- 

* « 

velopmental process. Additional evaltiation efforts are re- 
quired when the development of a product is cdsaileted to dem- 
OMtrate its .educational merif* ^ 
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ASSIJKP7S0HS 

Several assu2?>tioas underlie the evaluation nodel t?re- 

K 

sented in this paper* These aasun^^tions ehould.be considered 
before atteiroting to apply th6 suggested zDodel to any given 
effort of iixfinnzrfcior^ai deveiopmenr • .Vi^jlarliig' these assxifep- 



tic^s vouIg obviously lin^it the utility of the sodel for a 
spec^ific pxi^pose^ The model is based on five major ass^s3:5>- ^ 

tions. They are: . . 

^1). The developaent of instructional materials usDlies 
a long process of coordina.ter& efforts . This process can be 
shortened by efficient plsianing and coordination among its 
coEE>onents. However, 'jshort cuts" intended to skip signifi- 
cant stages of the developing^procfess xaay lower the^ quality 
^ of the product and in, the final analysis <io not shorten the 
aevelopmental process. The ^ucatiotial system mtist r&ali:2e 
that the devei^opment of educational mgans should be allowed 
to benefit from long-term investments, similar to those avail- 
able in industry o;r defence. Cocroared with the developmental 
processes in industry, which take adva;itage o^' the extejisive 
pool of Scientific knowledge, the process of development in • 



^ucation can rely only on the. limited knowledge of the b 



havioral sciences. ^usT'they can "Se^eS^eCtBd-^ take-^onger 
and be much more difficult than those In. Industry or similar 
areas. 

(2) The development of instructional materials must be 
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a coqpeljltivB^ask of a groap of persons yith a 'vide range of 
experti^^^ka n<|t an ixidlv±dxxit or sr*all group effort vrith 
1 i 7&i tgdfexpertise > An integrated tea^ of specialists in the 
siibject r:dpter, learning theory, teachii^ laethods^ new zaedii 
^valcj^^^n^and writing techniques should be involved in the de-- 
yelopaesit of^curriculun materials. Such a teaa should replace 
the^irad/tiohal 'tesct book writer* if sfgnigicant instrtxction- 
al ^la^Leri^ls are* to be developed* ^ , 

^ «) Fonaative evaluation should be* an integral part of 
the ^ev^iopsental ^ocess > This assuxrotion ha^ twQ itnportapt 
iE^Jcati<ms*^?ir8t, evaluation has to be conducted froa the 
b^in^i^g of the develorxaental process rather than at the end* X 
of tfiris pi^ocess. Secondly/ fonaative evaluation should sery6 

I ■ ' • • -y 

tbe n^ds of the develozsaentai teaia rather them assess its 
ef^fed veness for external audiences . Sf forts shdtzld be made ' 
tc reduce the threat of evaluation by d^aons tracing, its con- 
struc^Sive role in the deve lopsaental process, ^ven when sufch 
an ^f^>rt might result in the coaptation of €he internal eval-^ 
uJ^or and in a deoreaise in credibility £oz external audiences . 

^ 1 (4)* The evalifittion team or the evaltiation speciali*st^ 
sh6Ml not be the only ones involv^ in eva3Sa±Son activit- 
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fes within the develcpsehtai process/ ed^t^ugh ^;hey should 
bgar^^e gain r^^(xxsii>ijdty tar the, coMtSPi tt 6g ev^^ 
It i0 not feasible to assuiae that the evaluation team can 
se^ye all evaluati<»i ifeeds of^the deveio^menlf Waxa/^^nor is it 




desirable that evaluation be perceivedf as ^soset^lng ^hat ' . 



^ "beloags" only to t;he evaluation specialists. This assussJtion 
ic^aies that curriculum dey£liH>ers have to be trained in evalu- 
ation theory and techniques to gain basic ski Us and develop a 
positive- attitude toward evaluation. 

- Decision makers throughout all levels of the devel- 
opment organization or project should be coteitted to t he util- 
ization of evalua Uve infonaation in their decision -making pro- 
cess. This does not iiroly that all the decisions are based only 
on rational considerations, nor does it imply, that all recom- 
mendations resulting frra the evaluation must be adopted. How- 
evfer, decision makers should ^strive to increase the rationality 
of their decisions and be ready to justify those that cont?:a- 
dicf the evaluative information. In such cases the confronta- 
tion of the decision maker with the evaluation information 
might result in a significant contribution to the development- 
ai process, even though the evaluation recommendations are re- 
jected. No decision maker, ft s ready to admit that he will not . 
use the evaluative information unless it supports his decisions 
that already have been made. Too often evaiuatior/ is conducted 
when. there ^re no intentions to make use of the results. 

^ . AN OVERVIEW OP-THE EVALOATieU ifDDSL * ' 

• The evaluation model is based on experience in curriculum ' 
development which suggested that six major stages are involved^ 



in the development of an instructional unit. These 'stages are: 
1. The planning stage; ' ' *. 



2* The first draft .^tage;* 
3* The pre-trial edition stage; 

4. The eacperisiental edition stage; 

■ J 

5. The seai-final edition stage; and 

6. The^ final edition stage. 



Sacn developstentai stage is accoxmanied by evaluc^tion' 
activities as shcnm in Pigtire 1. 



Developmental Stage 



Evi3,uation Activity 



1 . Planning 

2. First draft i 

3- Pre-trial edition 

• 4. Experimental edition 

5. Semi-final edition 

6, Pinal edition ' 



Context evaluation 
Bvaltiation by xinit de- 
velopers 
Inspection by experts . 
Laboratory trials 
Field trials- 
Reptorting 



Figure 1: Developnental stages anji evalu&tion 

activities in the process of developing 
an instructional unit. 

The suggested developsaental' process has aj^'^ljtere.asiQg 
spiral structure, in which the transition frcA one stage to 
another is based on information provided by evaluation activ- 
ities. Each stage ^^^Ibdes^Tlev of curricrxlum mater- 
ials/ evaluation, and revisio^ of* materials for the next 
s[tage of development. The spiral nattare'of the noael in5)lies 
also that in certain cases it is necessary to repeat the same 
stage several times until the jnaterials are. ripe- for. the next 
stage of development. The "growirig" nattire of the spiral 
structvade is daaonstrated in Figure 2. The develojaaental 
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Final Edition 




Field 
Triar 





^^^^ f-, : >^ ^ -- * 

^^C.Jflhxrt^2l A Mdael far the Dcveloipent 
lI'^Rir' >nd Evalifittion .o£ Instrtictional 



Semi-final Edition 



Experimental Editioft 




ZLaboratory\ 
Trial \ 





process starts with a planning stage in which the development 
team clarifies its task and "structures the natujre of the cur- 
riculum \mit to be developed. -A first draft of the xmit is' 
then developed and . evaluated by the members of the development 
team to check its ^ngruepce with the rationale , underlying its 

"development . The first^draft is revised intQ^^ pre-trial edi- 
tion and submitted for inspection to a group of experts*. Based 
on the feedback obtained from the various experts, an e^tperi- 
mental edition of the unit, is developed to be "tt:ied ' out with 
students and evaluated under ladx^ratory conditions. A furth^^ 

. revision of the tinit is then introduced into a natural setting 
of the school and evaluated under conditions that hopefully 
approximate normal use of the unit. ' Based on the evaluation 
of this field trial a final revision of'the unit is conducted. 
The unit may now complete its entire cycle of development a]:id 
be ready for wide distribution and summative evaluation by po- 
tent?.al considers. The final edition oi the iinit is accomparv- 

ied by cin evaluation report ^which describes the entire devest- 

" ' ' ^ ' '. • ' 

opraentai effort and presents the evaluative data gathered dur- 

ing^i-the process of development.. ' - ^ . ^ 

The remainder of this paper is devoted to -.a description 
^ the evaluation activities at the Vfitrious stages of develop- 
ment^ problems involved in implementing such activities, and 
examples of some solutions to such probleias,* - _ * . 
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THE PLANNING ^STAGf ' , ^ 

* This stage starts, when the ^evelogaent team has been ' 
assigned to the task of developing a curriculum vljtit on a de- 

fined or semi-defined topic, At .the end of this ?tate. the \ ' '1' • . 

»- ' ' ^ '. 

topic of the unit has to be clearly defined^^.the s\d>ject mat- ' 

teir to be covered by the unit has to be. determined/ ^ii^truc- ' \ 

tional methods to be. applied in the unit have to be c^sen, the ^ 

general ' str\icture of the unit has to be dfefined, and a^c^orking 

plan for the team has . to be developed^ Decisions of the de-/ 

velopment. team a-t this stage are. facil^ated by various kinds. 

of ^^formatiori, such^ ^s information^ on, the needs of students 

for whoip the unit is bping ^develope'd/ general educational 

goals that must be served (e,g./ goals prescribed by'^a^anda- 

toxry syJlabus) / alternative teaching methods, and patterns of . 

instructionai^materials* Such information can be obtained • 

from the liter^p^ttire,* of f icial documents/ educational consult- 

ants^ or £rom' direct observations .in c^lassrooms and interac- 1 

tion with teachers and students*. Obviously, a systematic ^ ^ 

"needs assessment" study is desirable,, although in most cases . 

^ . ' /-^ ' ' * ✓ • ' :* . • 

it is" pot feasible to conduct such, a '%tudy^ at this stage and \ 

the* teajpDu-feas to -utilize existing knowledge 1 (or assumptions) ' 

regarding *the need$ .of * students and t^ach^g io be .seryied by 



inent/ as they were experienced ii**th9' NE*PA Prq^ct, included ' \v-. .^ 



..th& unit. . ... V V ■■' - ■ . ■ • -'- ' . .-^v 

Typidal evaluation activities at this jSt^ge'Qf develop- 



"^^Vhe following: ^ 




(1) Review of related literature- T]iis is anoint ef- . ' 
fort of the development team and th.e' eval^tion staff. The' 
development team focuses' mainly ons •documents related to the 
subject mattet of the unit while the evaluation .staff reviews 
research and evaluation^ studies or. related topics of interest, as 

-s^ s esthe i r findings and proyides summaries to the development 

team. ^ ^ . ' 

(2) Pilot observations • in classrooms; The .evaluation 
staff assists the development team in designing and conducting 
initial .observations of classrooms occupied by students of the 
target population.' . . ' . 

• (3) interviews of^xpeirts and practitioners. These in- 
terviews are -mainly conducted by tri^ development .team; however/ 
in' certain cases', a mqre structured interview .schedule may be " 
developed and implemented Jay the evaluation staff. • ® 

■Whenever feasible and desitable, two additionajL- evalua- 
tion activities should be implemented at t^his stag^. They are; 

(4) A systematic "needs assessment" study to delineate 
the needs of students and teachers ■ to be served by^; the unit. 
\ (5) An "input evaluation" intended to dev.^lop and assess 

new approaches to problems confronted in the cxirrl culum upi t . 

'. ' ' 

• ^Sek e.g., Ralph Hoepfner/ et al., National g ribritie^ 
for Elementary Education. CSE Monograph Series -in Evaluation,, 
Vol. 2, Center for the Study of Evaluation, University of Cal- 
ifornia, Los Angeles, 1973-. • ' ' • 

• ' ^ . ' ■ • ■ 

^Stufflebeam, et al., Evaluatiroir-and^ec ision 'Making , 

pp. 222-229. *■ ... ' , ; 't'^f ' 
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For this purpose the •eovoeat^ teaia* xaethodology can be util- 

T3Z FIRST DRAFT STA£S ^ 
This stage begins vhen -the devaiopsent teaffi has clearly, 
defined the topic of ^e tmit and subject xjatter to be in- 
eluded, th6 instructional siethods have been chosen, and a de~ 
velopaent plan has been approved • The tean is ready to start 
writing a first draft of the various parts of the unit, includ- 
ii^g' sti3dent .objectives, reading materials, practice exercises, 
lab experiaepts, and the teacijer taamual. The tean also starts 
developing supplej^^tary zaateriais for the unit such as xaodels, 
..Sjhartfs , glides , and filn loops 

* At this stage two main questions should he answered: 
' (1) Are "the materials being developed according to the 
plan proposed in the first stage? ^ 

» ^2) *Is the snit beix^g developed accordir^to the "phil- 
osophy", of the oroject?. 

To answer these questions the develojsaent teaiD can use a 
•^checklist" coiaprised of specific questions regarding the^ar--.^ 



ious aspect^ of the unit* Within the HBTA Proj'ect stich a 
checklist has b^en developed by the evalufttlon team pn the 
basis of some principles for the preparation and ade^ixtation 



Diane L. Reinbard/ •H^thodology peveloj^nsnt for Input 
Evaluat|.on Using A<|vocate Design Team»* Ph^D. disserta- 
tian, TheA)hio State tJni varsity, 1972* ^ * * 
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of iastrtictional naterials suggested by the project director. ^ 

The cdiecklist vas used by the deveix^paent tesn tiu:t»sbout the 

process of .developing tSe first draft of -the xmit in a atteirot 

to revise the draft in such a way that ziost of tiie questions 

incltided in ^e cbeckJLiiit could be ansverefi af f inaatively* 

The answers pd^ided by the develoj^aent team are obviously 

subjectivej^ the^fore^an additional attesspt to answer the saiie 

questions in a xaore objective way i's made in the next stage of 

• * 

the development* How^Sjpsr, it seems to be iniportant that a 

•self evaluation* be^ conducted at this stage as a basis .for, 

further evaluations and as an expression of the team's interest 

in the u^^ilization of formative e^^uatiqn. / 

Dxiring the first draft stage the laajor evaltxation activi-^. 

ties, are conducted by. the development team rather than the 

evaluation team. The major efforts of the evaluation , team at 

this stage are devoted to the develojsaent of an evaluation de- 

sign for the projects The e valuators, have to interface with 

the developers to delineate the evaltiation needs of the tsext 

t 

stages of the deVeioj^ient process. The evaluation design and 
its instruments have to be developed in such a wa^ that the ^ 
evaljiation need* of the project are adequately served. ^ cosir 
xaon pitfall in the evaluation design is the use .of zaeasureaent 
instruments based on their availability rather than oil the 



' . evaluation needs of the project. ^ • . f ^ :--^j*'4;ls 

' - ^' " ■ '~ ^ ' *- ^ , "^-7- -55^5 

. ' . -* Moshe Snilans}^/ "Assuu^tions for Preparation .and Adapt- . . . .i^ij 

I ' . ' ation of Curricnla. and M?^a fbr. the Culturally. Di^adTantaged." _ , ^w-, 
-/^■i^- iflje HS2A Project, 3fel-Aviv- Osiversity, 196€^.(mae6.>., . * * ..-^^ 

?te-^--%f v^- . — ■• r : . ... s-^-^'-^^i^a^. 
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T3Z ^ZS-TSlMa SDITICa? ST&S3^ 
T hi s stage sVaxts vfaenever tiie de:^2xjpsneBt fca&a .has com- 
pleted the revisiot of the first draft of the instriKtioaal 
uait a^:9rding to^tire results of tbe "self evalijaticm* pieced- 
tire described in the previoos section*. The unit nbv per- 
ceiv»ed /3fy^&e. teaa as being congruent vith its plans and prin-- 
ciples tmderlyicg its developa^t* It is now tiae ±0 determine 
how the imit is perceived by others and to ysbai, extent it can 
be fuafther in^^roved prior to its trial ixi e^cperintental classes. 
Many cxirriculum deyelooers skip this st^e of develc^asaent as 
they 4re eager to introduce their products into s<diool classes 
without any further delay. Such pr^siature field rtriais should 
be discouraged. It is our experience that smch can be learned 

Z * * ' ' 

ab^ut an instructi<mal unit through its inspection by a panel . 

of experts. The feedback provided by .such earoerts can'^be such 

Hore valuable for the icgroventent of t2ie unit than 'the infoiJi^- 

ation obtained -^roa a pr^atu^e field trial. * - 

•The fallowing guidelines aay be used in this regard: (a) 
• ♦ ^ ' • , 

The panel should include exiterts in various areas relevant to 

the development of the instructional' unit' such as sublect laat- 

V ' • I - ^, ^ 

ter, teaching laethods or instructional iiedieu (bKAithoiMh ran^ ' . T*I5 J 



dra selection of^^ experts is not n^sss^arjC it is i3^rtan|^ to _ .-^ ^ 



awid a biased ^casg^B of /ecpefts 5?hose jo^ynions, ze^ExSiag ' tte T^.^L. --'3^ 
tadt are known in advance, (c) , Tise^ea^gi^s. v6bi^d>e^>is3^ 
in^^qt the aat^als..of.the tmi%^ la^^^^s^^^^ 




spoad ofily to the qoestioa *Khat do ycm thiiiX abotrt this un^V* 
is not advisable. A detailed 'qiisst-ioiiixalre should be developed 
for this purjx>se; the checklist tised for self evaluatic«i m the 
. previous stag^ can be used as a basis for developiiig the ques- 
ticnaaire. (d) !Zhe nunibe^ of eaa>erts required for the iuspeo- 
tlon should be detencihed on the basis of the coiroie>dty of the 

Ins true ticaial unit and the, availability of time and 'xeaourcdl 

V 

for thrs purpose^ Frona oxir e2Q>erieiK:^ey five to ten experts 
were sufficient. 

Tt isdqbt also be appropriate at this stage to condxict soae 
ssiall-scale e>q>eriaents to test specific aspects of the mit. 
These expe risen ts should test feon^ponents of the unit such as 
unique teaching methods/ innovative media or special technical 
devices incltjded in ^e unit. 

The major responsibilities of the evaluation team during 
the pre-trial stage are': (a) assisting the develojraent team in 
selecting a panel of experts; (b) analyzing experts* responses; 
(c) developing meeLSuring instrtsnehts for the subsequent stages; 
and (d) conducting small-scale controlled experiments. 
THE EXPEICmSNTAL EDITION- SS&<S 
Based on the feedback obtained f roca the evaluation at the 
previous stage of develojsaent/ another version of the unit is 
now developed. This is the experimental edition of the unit 
to be tried out vith students and evaluated under laboratory 
conditicms. By " laboratory conditions* we refer to a con- 
trolled setting providing optieftal conditions toz JLxg?lesientatiQjL 
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^rsts pf the qnality pf physical resottrces/ teaching per- 
il/ iaotivation of teachers and students / and ar^ropriate 2 
adninistratiye support. Such conditi<His are usxaally not re- 
^ pr^entative of the •real" conditions available in the educa- 
tional systea for vhich the unit is being develc^ed* Eovever^ 
at this stage the questicai is i»t whether the unit can be 122- * 
pleaoifced in an 'average <5lass* but whether it can be iirole- ' 1 

jaented at aill. 

The value of the experiiasnta^ tryout of the unit should i 
not be underestiaated nor should itf be overestimated • Its 



value is overestisaated if ve try to assess ^the effectiveness , 
of the unit for otxr target poptilation only on the baisis of its 
performance under nonrepresentative conditions . It is xmder- 
es t isiated if we •fail to recognize its valxie in providing iza- 
portant feedback for the isrproveraept of the unit in spite of 
the nonreoresentative conditions <of the experiiaental trial* 
The nt2i2ber of classes ^to be included in the experiaent 
should be relatively small. It is de^enained laainly by the 
ability of the developiaent team to assure optissal experiment- 
al coiiditions in the participating classes concerning teacher 
training r technical facilities/ and intensive interaction 
wit6 the pxp^iiaental classes • 0txr experience ^owed that 
five to six classes 'are stiff ici/e^ eLtthis shsLge of deyelopr 
laent. The introduction of^Hte ^q^erija^^sH edi^pnhf^tbe 
unit into a Icur^e saspie of classes, sdghfc 'lifli^ th& ^ilit^ 





,5 3^ ^:^-*--^V>^^ 



the evaluanve infcnLation rhat it cas: provide. 

Teachers sno-ld be recruited or. a vol'JLntar'y r^asis 
secure hich quality ir.st.ructicn for the experxnter-t-a! 
of th^unit. Criteria for choosing teachers should be or. the 
besis of coES>etei:c/ / ceach:j:g exx>erier.ce , flexibility ar.d cosj- 
siitoert ro the experiJi^ental effort. Teac^^ers srhculd be pro- 
vided with exter.sive trair.ir.c ir. use of tJ'.e ur.it ir. ^heir 
classes, arid should be recuired to utplerxe.'-.t it ks prescrii^ed 
by tr.e developn^ent ar.d evaluation t^eains-^ever. if tr.ey disagree 
witXi scene of zhe ziefnod^ suggested m the teacher's nianual. 
Their reservarions should be considered m revising the expe:^ 
rsent^l e&ition for the next stage of the deveiopsent; however, 
at this stage the unit sjust r>e used precisely as intended by 
it-s designers. Instructional developers are sonetiiries re- 
iuctauit to "force" teachers to follow their instructions to 
the letter, especially when the instructions are related to a 
xinit which IS intended to prosote flexible and innovative 
teaching methods. A certain aiaount of rigidity is required to 
secure precise iisp lenien^tiorxrQf a new teaching cethod ±n or- 
der to be able to evaluate it adequately. 

The laa^or evaliHttioA activity during ther experiiaental 
stage should focus* 6n the observation of the process of iiq^le- 
oehting the unit in the experimental classes. Several obser- 
vation techniques can be used. One possibility is to develop 
a •scenario" of the unit which provides a detailed list of all 
the planned activities of the unit, their suggested sequence* 
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amd rinie est ir>at^ . Tr.e sceriaric is develcpec by 'ti:e evalua- 
tor after studying the instructional j^nit ar.d its supplement- 
ajTv saterials. Interfacmc wi'th tne developers he would vali- 
dat.e the scenario and after several revisions he sught end* up 
with an observation fom which could be used for classrco:^ ob- 
ser%*ations. Csing the :.ns tr^uzient m experurien tal classes will 
enable a ooctparison betrveen plaru^ed and actual activines, and 
VI 11 provide val-uable information m assessing the feasibility 
of u^Dlenentmo t-ne unit as intended. 

In namy of the nevly -developed mstuructional znaterials, 
efforts are zaade to 'breax" tJ^e traditional "tal): and chalk* 
teacnm^ n^tnoc s^y mtrocucticn of various mcepenoent worx/ r - 

study activities of individuals and groups of students as veil 
as other n^ethods of active student mvolvenient . In such cases 
It nj.ght be interesting to deterrrxne hov such ti^ne is spent m 
the experiiTtental class on active student participation as op- 
posed to passive listening. Suppose that a class of 30 stu~ 
dents IS participating in a 45 minute lessor^. This caxi be 
thought of as 1, 350 student-niinutes that can be utilized m 
one way or another* A simple observation instrument in the ' 
form of a matrix having a list of possible student activities 
in rows and tirse '(1 to 45) as columns will enaile a trained / 
observer to record the time devoted to each kind of activity. 
In our project we successfully used such an instrument for 



8 

e.g., individual work, grotro activity, talking to class, 
iist^ing to teacher, working individually with teacher, etc. 
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Vs^^r^cordmc xs:e r. jrnber cf studer.t-s ir. each act.ivit:%* z>et sur.ute. 
Ar tJie end cf rr:e cbser-^atior. ir.e total nini>€r of student 
rLLnxrces is found for each activity and a percentage is calcu- 
lated fros ti*e total nun±>er cf sfjdent i:L:.nutes m a given 
class, (in cur exaisple, 1,35C . This sethod ignores the con- 
tent cf the lessen cut ^nacles tne distinction oerweer classes 
tJvat practice sore student invcl vecrent and these using sere 
^ *" r a d 1 *" * c * * Ti*^ " - ^ 



A third ntethcd cf ccser*/aticn can be a nonstr jctured ob- 
ser\'aticn cf unit developers m the experimental classes. Al- 
tnouch t^s IS not recoxxaended as the -only iikethod to be used 
for^a ctvert unit, it is very usportant that any syst^atic ob- 
servatlc^n- .^conducted by the evaluators of the project be supple- 
n^ent^ b>^ such "non-sys ten^tic*' and sub3ective observation con- 
ducted by those who developed ' the instructional laaterials and 
nave tc ii^prove then. The insights that laay be ^erived from 
this ".ethod are sonetizies zuch more valuable than those frcxn 



the other sethods/ aind therefore should not be neglected. 

o 

Standardized observation schedules can t±sp be used to' 

<^^erve the activities in the experimental clasps. These 

should be used only if they measure variables relevant to the 

evaluation of the unit. Too often, standardized instruments 

are used "just because they are available" and actually do not 

« 

make any significant contribution to the development of the 



9 

See for exairole: Simon, A. 6 Boyer, E.G. (Bds.)^ Mirrors 
for Behavior; An Anthology of Classroom Obseirvation Instru- 
ments . Philadelphia: Research for Better Schools, 1968* 
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insfcractional tmit, 



The evaluation plan for the escperiiaental stage should al- 
SO AnclTjde sceae seasur^ent activities intended to assess st^ 
dent aciiievesaents . Standardized achievejuent tests can be used 
provided that they are related to the instructional objectives 
^of the unit. However , In zsost cases special tests have to be 
developed to provide for meAsuresient of the achievement of 
specific objectives that have been defined for the tinit. Such 
tests have to be developed following £he standaxd orocediire 
for test developsaent including iten analysis which xnust be 
based on data obtained frcsa students who hss^e already studied 
the material covered in the test. Thus, the experissental 
stage is used not only for the trial of the instructionai xaat- 
erials but also for trial of achiev^ent tests which are to be 
ijfeed for the evaJLuation of those materials. The resxilt of 
this is that one is actxially trying to evaitoate an incosaplete 
instrtictional unit by means olE an incoirolete achievement test. 
^Therefore, only limited conclusions can be drawn from such stu- 
dent achievement data* Further investigation in this area 
should be conducted in the next stage following the validation 
of the newly-developed tests. 

THE SEMI-FINAL EDITION STAGE 
On the basis of the e'"^aluation of tfae e3q>erimental edition 
of the instructional unit under laboratory conditions a fiirther 
revisioii of the unit is being ..de^^Cc^ed in tbis.^^^ge;i^ This 
seai-final edition will be introdiwed ;at this stag* into a 



y 

,naturcil setting of the sci^l and evalnated under <xmditions 

that hopefully approxiraate normal use of the unit. Tvfo laajor 
questions have to be aiiscered by tixe evaltaation at this stage. 
Siey are: ^a) Is, it feasible to . iiroleaent the suggested unit 
in a regular school j^ithout providing special conditions -such 
as those that vere provided at the experisent stage? and (b) 



Vfhat isqpact does the unit Save on *the achieveaents of students 
on regular classes of the target population? to answer these 
qxiestions a field trial nrust be ciSnducrted, The laajor differ^ 
^ences betveen the field ^ial and the laboratory trial can be 
siTT7nariz6d as follows^ ' • 

(1) Teachers should be selected on the basis of ^ random 
selection procedtire rather than on^a voluntary, basis in order 
to enstire a laore representative santple of. the total teacher 
pc^tilation for whon thie instructional unit is_ being ^veloped. 
Teachers who voliaiteer to paxticipate in- the trisfl'^of sosie 
innovative effort laight be significantly different frc^ the 
"average* teacher regarding known and utiknown characteristics. 

(2) No special training or guidance should be provided _ 
to teachers participating in the 'field, trial excep€ f or the 
instructions provided JLn t^e teacher 'gjnanual pf. lie unit or 
thp kind of training prescribed by the unit for Regular {not . ^ 



ea^erlTnen tap^teacdieags ^ ^ - = . ^ ' \S . ; r- M^i0^ 

<3) A considerably larger sam^ of clais^s. should b^ .^.n^"^^""^ 
used for the field trial to obtain a reasonable^ r^resentatioii \.^v., ..^ j 
p£ the taxg&t population. However, one.jB&omd, iaeaa^^in jaiiid / ; 
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^ =- — ^~ that the r^resentation -of a sample is not determined only by 



its size bttt aXso by the procedtire used for aravring the saiapie. 

Tiros, a small randoa sairpXe sight be sore representative thss(^^^ 

a large biased srnnple> 

(4) The evaluation of student achieveaents should be done 

by seans of the tests which were developed and validated at the 

previous-^^rt^afge . 

15) Althdtigh the major function of both the fields and 

• - * 

laboratory trials is formative rather than sruscaative/ the po-- , 



ten tial^. utilization of the, field trial for siiisaative ptirposes 
is obvious. However/ since the evaltiation at this stage is 



condticted by an internal evaluator whose credibility for exter-- 
nal atidiences laight be limited, the involvement' of some exter- 
nal independent evalxxation agency in the process of the field 



rrial is strongly, recorrmended^ Such an agency siK>uld serve as 
a secondary evalx:tator or meta-evaltiator to increase the credi- 
bility of* the evaluation in deiaonstrating the merit of the in- 
^ structional materials ^or potential cli^ts* 

THE FINAL EDJTION ^STAGS' ' 
The senii-^inal edition i^ now revised into a finauL edi- 
tion on the i>aSis of the results of the fieid trail*. One 
should ei^ect that if the. instructional unit went thrjbugh the 
^ * / who]^ process of develo^aent ^md evc^uation^ only/miiior ^ 

1 changes would need to he introduced at liis"s€age*. However r . . . J^V^' 

. should the results, of the field trial suggest jaajor revisions . v j 
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trial of the revised semi -final edition zai'ght be advisable 
prior io the es tab lishsient of the final edition. . 

When .the f inpl edition is cozspleted/ the instructional 
unit is, ready for cozasaerciai production and dissemination to 
potential clients. The final edition of the unit should be . 
^LCCoz:?>anied i)y an extensive evaluation report which docusients 
the entire process of 'developskent and provides ^^uatiW ^ta 
regarding the educational merit of the unit* Thus, the major 
task of the evaluator at this stage is the development of such 
an evalxiation report. The evaluation report should concen- 
trate on the results of the field ttial/ but it shoxild also 
describe the eairlier ^evaluation activities as a d^onstration 
of the fonaative evaluation process. The evalxiation report 
should point -out the strengths as well as the weaknesses of 
the instructional materials amd establish the limits of their 
usefulness by a complete descri^ption of the population on which 
it was used, 'if external evaluators participated in some way 
in the field trial, their assessments should also be included 
in the report. The style of .the report should not be^^too 
technical, but at the same time it should include detailed 
data that can be reamalyzed by other evaluators* This might 
be accon5>lished by the incliision of^a naurrative text with a 
few major statistical tables, as well as a more detailed pre- 
sentatioi> of statistical data in the form of an appendix. 
SCTje potential constWrs of the instructioxial unit laight be 
interested in an independent stsmaativa eva^.uatix>n of the' 



unit prior to its adoption in their school systeias; the in- 
*temal formative evalxiation should provide a solid basis for 
the conduct of such further evalxiation/ 
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